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METHOD FOR GENERATING 3D COMPUTER GAMES. AND 
COMPUTER GAMES USING THE METHOD 

Field of the Invention 

The present invention relates to the field of image processing. More 
particularly, the invention relates to a method for generating computer 
games, to computer games using the method, and to display means for 
playing the game. 

Background of the Invention 

Computer games are widespread nowadays. They typically employ a bank 
of computer-generated images and of sounds, which are either recorded or 
computer generated, and which are combined according to predetermined 
rules, to produce a sequence of actions related to actions taken by the 
player. Thus, interactive games employ the power of the computer to cause 
a specific occurrence, or a specific series of occurrences, to take place as a 
result of input provided by the player. 

The technology employed in the art suffers from several severe drawbacks, 
the most important of which being: 1) that there is only a limited feeling of 
reality, no matter how high the quality of the image; and 2) more 
importantly, the cost of generating computer images is very high. 



WO 98/45004 PCT/IL98/00137 

-2- 

Even in some instances, where computer-generated images have been 
substituted by real shots, the feeling of reality is minimal, because no 
depth of the scene is obtained, and in this respect the situation is even 
worse than with computer generated images, where 3D-like images can be 
produced. Furthermore, only a small number of shots can be used, because 
of image processing limitations, which affects the feeling of continuity of 
the game. Above all, however, the existing computer games employing 
real-life shots are uninteresting, because they lack reality and 3D feeling, 
and because they are made of a narrow choice of repetitive sequences of 
events. 

In a copending patent application (WO 95/33340), the description of which 
is incorporated herein by reference, several considerations pertaining in 
general to the generation of 3D-like sequences of images are detailed. These 
are further briefly discussed below, for ease of reference. 

There are several known methods to perceive depth. Examples are 
stereoscopic perception of depth, psychological perception of depth using 
visual cues and the use of changing focus to perceive depth. A viewer 
viewing a scene in which the focal point changes will perceive that the scene 
has depth. 
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Stereoscopic depth perception requires that the viewer views a scene from 
two different locations at the same time. In real life, this means that he 
sees the same scene using both his left eye and right eye, each eye 
perceiving a somewhat different image. When two dimensional visual 
images are substituted for real life, it requires the use of two images taken 
from different right eye and left eye locations respectively. 

On the other hand, a viewer can perceive depth by virtue of a number of 
psychological depth perception cues which are present in both real life and 
in two dimensional visual images. Common cues include effects where the 
change of coloring or softening of the shapes gives a feeling of distance to far 
objects, distribution of light and shade on an object, overlapping of contours 
as when a near object hides part of a more distant object, geometrical 
perspective for example in the case of the converging of the parallel rails of 
a railroad track as they extend in the distance, movement of an object in a 
visual image either towards or away from the viewer or audience, and 
movement for stimulating the Pulfrich effect 

This is true of all the various devices in which two dimensional visual 
images are projected onto screens. Such devices include televisions, motion 
picture theaters, training simulators, multi-media systems and so on having 
applications in the areas of entertainment, work, education, exhibitions, 
training, etc. 
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A number of devices had been proposed to permit a viewer or an audience to 
view three dimensional visual images or at least the illusion of three 
dimensional visual images when viewing two dimensional visual images. 
These broadly fall into three groups: those which provide stereoscopic 
viewing, those which help create the psychological illusion of depth and 
those which display images on a three-dimensional display system, such as 
a surrounding screen. 

Devices providing stereoscopic viewing generally require the projection of 
two pictures of the same view, taken simultaneously by more than one 
camera from slightly different angles, to emulate the binocular vision of 
normal eyes. Several examples are shown in U.S. Patent Nos. 4,420,230 to 
McElveen, 4,714,319 to Zeevi, and 5,225,861 to Marks. The use of a 
vibrating mirror or screen is shown in U.S. Patent Nos. 4,130,832 to Sher 
and 3,814,513 to Nims et al. Other systems require special spectacles to be 
worn by the viewer or other devices which direct the proper view to the 
proper eye. All these systems are cumbersome and costly to use. Also, they 
tend to cause an uneasy feeling in the observer often resulting in headaches 
that may become severe. Thus these systems have failed to become popular. 

The second approach is shown in U.S. Patents 4,651,219 to Rickert, 
3,582,961 to Shindo and 4,000,367 to Field wherein depth perception is 
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provided by isolating the image to be viewed. In this way, the stereoscopic 
cues of the space surrounding the image are suppressed so that the inherent 
psychological depth indicators in the picture can be sensed by the observer 
who perceives a two dimensional visual image as a three dimensional scene. 
Other devices which produce a similar result are described in U.S. Patent 
No. 3,820,873 to Redington et al., U.S. Patent No. 4,154,514 to Harvey 
which utilize a curved screen, and U.S. Patent No. 4,941,041 to Kenyon 
which utilizes the Pulfrich illusion. 

Other devices for providing three dimensional viewing are also described in 
the following U.S. Patents: 

U.S Patent No. 2,468,046 to de los Monteros discloses an apparatus for 
projecting and viewing images with a depth effect through the use of at 
least one mirror to reflect the image from a projector onto a screen such that 
a degree of displacement between the images produces the illusion of 
stereoscopy. 

U.S. Patent 3,514,871 to Tucker discloses a wide angle visual display for a 
training simulator which eliminates distracting discontinuities found in 
many wide angle displays. It does so by providing three virtual image lenses 
between the viewing point and the two dimensional displays and by 
providing substantial duplication of the projected computer generated 
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image on the marginal portions of the displays. This creates binocular 
vision at the corners of the displays which gives the illusion of three 
dimensionality. 



U.S. Patent 5,274,405 to Webster describes a device into which the head is 
placed wherein images are projected both to the area of visual attention and 
the area of peripheral vision of each eye. The images are focused on the 
screens to provide sharp stereoscopic viewing throughout the field of view. 

Another technique which is also known in the art is to take pictures of an 
object, e.g., a room, from different points of view, and to generate therefrom 
sets of spatial parameters, or database for novel view synthesis. Such a 
method is described, e.g., in "S. Avidan and A. Shashua, Novel View 
Synthesis in Tensor Space, Hebrew University Technical Report, 1996" (also 
available from the World Wide Web at http://www.cs.huji.ac.il). This method 
permits to generate three-dimensional images of the object, and to generate 
new images extrapolating from the parameters so obtained. This method 
(called hereinafter "the parameters method", for the sake of brevity) is a 
suitable method also for generating games according to the present 
invention, as will be understood from the description to follow. It suffers, 
however, from the drawback that substantial computational power is 
required to carry it out. Nevertheless, as less expensive computing power is 
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made available to the public constantly, the aforesaid parameters method is 
a suitable candidate for the purposes of the invention. 

Other methods for producing three-dimensional moving images are based 
on computer generated images. An example is U.S. Patent 5,184,956 to 
Langlais et aL, where computer generated data is used to produce a three- 
dimensional display for the purposes of driving training. It is easy to 
appreciate by any one skilled in this field that it is not difficult to adapt 
computer generated imaging techniques to many types of display systems 
such as stereoscopic display, wide angle display, and also three- 
dimensional display. This is so because the entire information required of 
each part of the image is known to the computer's program - including all 
the three-dimensional structure of each object, its surface color, texture, and 
so on. This seeming advantage is also the major disadvantage of such 
systems: the preparation of a realistic image requires painful and elaborate 
design of each and every detail. Thus the display is either non-realistic or 
else extremely expensive and laborious. 

The method for generating a composite, three-dimensional-like image 
sequence according to WO 95/33340, by the assembly of two or more 
separate image sequences, comprises the steps of: 
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(a) providing a main sequence of images; 

(b) providing at least one secondary sequence of images the contents 
of which are at least roughly related to the contents of said main 
sequence; 

(c) selecting the positioned relationship of each of said at least one 
secondary sequence of images and of said main sequence of 
images; 

(d) processing each of said at least one secondary sequence of images 
so as to optimize its matching with said main sequence in the 
selected positioned relationship with it; and 

(e) displaying said main sequence of images and said secondary 
sequences of images at the same time and in substantially the 
same positioned relationship as selected in step (c) above. 

The term "three -dimensional-like" as used herein (also referred to herein as 
"three dimensional", for the sake of brevity), is meant to indicate a sequence 
of images which is displayed in more than one plane, and which thereby 
evokes the feeling of real action taking place around the spectator. This 
feeling, more fully described herein, gives the viewer the feeling of being "in 
the movie", and provides a sense of reality unattainable by regular movies 
which are displayed in one plane only, as hereinafter defined. The three- 
dimensional-like image sequence of the invention is "composite", inasmuch 
as, in order to obtain the said effect, more than one image is displayed, as 
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said, on different planes, the resulting whole image being composed of said 
different displayed images. 

The term "plane", used herein in relation to the display mean on which a 
sequence of images is viewed, is to be understood to include display surfaces 
in general, even if they are curved to some extent, and the term "plane" 
should not be interpreted as meaning that concave or convex surfaces are 
excluded. Similarly, when reference is made to an image which lies 
substantially on one plane, this should be understood to include also an 
image which is displayed on a curved display, as explained above. 
Furthermore, it should be understood that it is possible to use curved 
displays essentially made of one piece, onto which a main sequence of 
images is displayed, along with one or more secondary sequences of images, 
each sequence of images occupying a different portion of the curved display 
or screen. As will be apparent to a skilled person, such a single, curved 
display is equivalent to a plurality of single displays, each used to display a 
single sequence of images. Furthermore, such a single curved display would 
necessarily be made of sections which substantially he in different "planes", 
as defined above, when using curved displays. 

The main sequence of images is preferably, but not limitatively, located in 
the central part of the composite displayed image. This is because the main 
image sequence is the standard by which the secondary image sequence is 
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processed, and it will further be, in many instances, of higher quality (in 
terms of resolution, number of frames, etc.) than the secondary image 
sequences. However, it is permissible to position the display displaying the 
main image sequence at some other position relative to the center of the 
picture, if technical or aesthetic requirements demand it. 

Once the position of the main image sequence is selected, it is necessary to 
select the location of the secondary image sequences within the composite 
picture. There is no theoretical limitation to the location of the secondary 
image sequences, and some of them can be located, e.g., above or below the 
display which displays the main image sequence. (These alternative 
positions are not shown in the appended drawings, for the sake of brevity, 
since they are self evident from the description provided for the illustrative 
geometries). Furthermore, a secondary image sequence may border with a 
second secondary image sequence, and not with the main image sequence, 
whereby said second secondary image sequence borders, e.g., with the main 
image sequence. However, for most applications, when it is desired to 
simplify and reduce the cost of both the process of creating the three- 
dimensional image sequence, and of displaying it, it is preferred to position 
the secondary sequences beside the main sequence. 

As stated, in order to provide a three-dimensional-like image, it is necessary 
that different image sequences be displayed on different planes. Typically, 
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the display on which the main sequence is viewed is positioned at an angle 
with the one or more displays on which the at least one secondary sequence 
is viewed. The term "positioned at an angle", however, should be interpreted 
here to indicate that at least a major portion of the secondary image 
sequences lies on planes (as hereinbefore defined) which intersect the plane 
on which the main image sequence is displayed, no matter what the actual 
interface between the display means of the secondary image sequences and 
of the main image sequence are. Thus, for instance, curved displays may be 
used, if desired, where the transition from the display of the main image 
sequence to that of the secondary image sequences is smoother than when 
using two fully planar displays, positioned at an angle with respect to one 
another. 

As stated, the secondary image sequences are processed in order to provide 
a matching with the main image sequence. Therefore, according to one 
preferred embodiment of the invention one or more of the secondary 
sequences of images has been recorded at a time different from the time at 
which the main sequence of images has been recorded. 

One or more of the secondary sequences of images may have been recorded 
at a location different from the location at which the main sequence of 
images has been recorded. 
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Furthermore, one or more of the secondary sequences of images may be 
computer generated. It is also possible to mix images of different origin. 
Thus, for instance, in one particular embodiment of the invention some of 
the images in a sequence of images are computer generated, and some are 
images acquired by movie or VCR equipment. 

According to a preferred embodiment of WO 95/33340, the main sequence of 
images and the secondary sequences of images are recorded on different 
media. Typically, but non limitatively, the media on which the sequences of 
images are recorded is selected from among film, video and digital data. 

In many cases, and particularly when different recording and/or displaying 
methods are used for the main image sequence and for the secondary image 
sequences, the number of images in a first sequence, for a given sequence 
display time, may not match the number of images in a second sequence to 
be displayed together with said first sequence, for the same display time. 
According to a preferred embodiment of WO 95/33340, therefore, the 
method further comprises matching the image sequences to be displayed 
together so as to reach essentially identical starting and ending times of 
display for all sequences. 

As stated, the secondary image sequences are processed to optimize their 
matching to the main sequence of images. According to a preferred 
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embodiment of WO 95/33340, the processing of a secondary sequence of 
images comprises matching the field of view of said secondary sequence to 
that of the main sequence. According to another preferred embodiment of 
WO 95/33340, the processing of a secondary sequence of images to optimize 
its matching to the main sequence of images comprises matching the 
apparent motion of said secondary sequence to that of the main sequence. 
As will be apparent to a person skilled in the art, the term "apparent 
motion" embraces two different types of movements: the first, is the 
movement that the camera appears to make relative to the scene being 
recorded, as viewed by the observer. For instance, if the camera rotates 0.3° 
per frame in the main sequence, to concentrate on an object that was 
previously at the edge of its field of view, the images in the neighboring 
secondary sequences must likewise rotate, at the same angular rate. 

The second type of apparent motion is that of an object within the image 
sequence. If, for instance, there is a road extending from the main sequence 
into a secondary sequence, and a car travels on that road, it cannot appear 
to change speed abruptly when it reaches the edge of the main sequence, 
but its movement must continue smoothly on the display of the secondary 
image sequence. However, in some cases less accuracy of transitions from 
main to secondary images may be sufficient. For example, for scenes which 
comprise many objects (more than 100), accuracy and smooth transition is 
required for the ensemble of all objects in general, but less accuracy may be 
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sufficient for every individual object of said ensemble appearing in the 
scene. 

In yet another preferred embodiment of WO 95/33340 the processing of a 
secondary sequence of images to optimize its matching to the main sequence 
of images comprises matching the displayed action of said secondary 
sequence to that of the main sequence. 

As will be apparent to a person skilled in the art, other parameters of the 
image can be processed to improve the matching of the image sequences. 
For instance, in a preferred embodiment of WO 95/33340 the processing of a 
secondary sequence of images to optimize its matching to the main sequence 
of images comprises matching the colors and shades of said secondary 
sequence to that of the main sequence. 

One advantage of the invention described in WO 95/33340 is that it is not 
necessary to provide in all cases image sequences of high quality and 
resolution. According to one preferred embodiment, for instance, the 
resolution of the images in a secondary sequence of images may be lower 
than the resolution of images in the main sequence of images. 

It has now been found, and this is an object of the invention, that it is 
possible to generate computer games which exploit the three-dimensional 
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like effect obtained by the method described above, and which lead to 
substantially improved visual quality and lower costs. 

It is an object of the invention to provide a method for generating 3D-like 
computer games, which employ real shots. 

It is another object of the invention to provide computer games which 
employ the method of the invention. 

It is a further object of the invention to provide means for playing the 
computer games generated according to the invention. 

Other objects and advantages of the invention will become apparent as the 
description proceeds. 

SUMMARY OF THE INVENTION 

The method for generating a composite, three-dimensional-like image 
sequence, by the interactive assembly and processing of two or more 
separate image sequences comprises the steps of: 

(a) providing a bank containing a plurality of individually selectable 
main images or main sequences of images, said images being 
images of true-life origin and said images or sequences being shot 
separately, or extracted from one source; 
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(b) providing at least one bank of secondary images or sequences of 
images, said images or sequences being shot separately or 
extracted from one source, the contents of which are at least 
roughly related to the contents of said main sequence, said 
secondary sequences being selectable as a function of the selected 
main images or sequences of images; 

(c) selecting the positioned relationship of each of said at least one 
secondary images or sequences of images and of said main images 
or sequences of images; 

(d) pre-processing each of said at least one secondary image or 
secondary sequence of images so as to provide a desired processed 
image; 

(e) providing selecting means to select a main image or a main 
sequence of images, as the result of an input provided by a player, 
wherein said selected main image or main sequence of images is 
displayed together with its related secondary image(s) or 
sequence(s) of images, at the same time and in substantially the 
same positioned relationship as selected in step (c) above; 

(f) providing means for on-line processing said main image and said 
secondary image, as a function of an input provided by a player; 

(g) optionally, playing a sound related to said main sequence of 
images and said secondary sequences of images, and/or playing or 
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generating additional outputs related to such sequences of 
images; and 

(h) optionally, displaying at the same time non-real life, computer- 
generated images in superimposed positioned relationship with 
the main or secondary images. 

The pre-processing referred to in (d) above may be effected to obtain many 
different results. For instance, pre-processing can be aimed at optimizing 
the matching of the image with said main image or main sequence of 
images in the selected positioned relationship with it, or to create a 
database of image parameters, or to change one or more of its 
characteristics, etc. 

By "images of true-life origin" is meant to indicate images which have been 
recorded with a camera or the like recording means, and which are not 
drawings or computer-generated images, but which may have been 
processed, or combined with other images, or drawn on. The bank of images 
contains shots taken by the various cameras, which are organized in 
sequences, but which can also be utilized singularly. Thus, for instance, 
when it is desired to generate an image on a side screen, or as a result of a 
joystick movement, e.g., zooming in, it is possible to utilize an out-of- 
sequence shot, taken from the bank, whenever this is suitable to create the 
desired result. Various combinations of shots and sequences are of course 



WO 98/45004 PCT/IL98/00137 

-18- 

possible, as will be appreciated by the skilled person, and such options are 
therefore not reviewed herein in detail, for the sake of brevity. Thus, each 
image stored in the image bank has, associated with it, a set of parameters 
which define it, in terms which are related to the type of game being played, 
and which permit easily and quickly to locate it, when needed for a given 
purpose, within the image bank. The number and nature of the parameters 
used to characterize an image may vary from one game to another, and is 
solely dictated by the needs of the specific game. Thus, the selecting means 
used to select a given image of sequence of images, may comprise database 
managing software, used to locate and retrieve images efficiently. 

By "extracted from one source" it is meant to indicate a procedure whereby 
a broad view movie, which has been shot with a wide lens, or which is a 
combination of two or more movies matched together using image 
processing and/or mosaic artwork techniques, is used to generate therefrom 
separate images or sequences of images. In some cases matching may be 
implemented without image processing. 

According to a preferred embodiment of the invention, the main image and 
the secondary images are processed, as a function of an input provided by a 
player, so as to zoom in or out of the original image stored in the image 
bank. According to another preferred embodiment of the invention, the 
main image and the secondary images are processed, as a function of an 
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input provided by a player, so as to simulate the turning of an observer's 
head. 

Other types of image processing are of course also possible and desirable. 
For instance, in one preferred embodiment of the invention the main image 
and the secondary image are processed, as a function of an input provided 
by a player, so as to change the speed of movement of the observer. 

Other types of suitable processing includes the generation of images by the 
parameters method described above. Thus, according to this method, 
whenever relevant parameters such as location, view angle, zoom, etc. 
(referred herein sometime also as "the observer") change, a new image is 
generated using the parameters previously made available by the analysis 
of the real life images taken. 

The additional outputs related to such sequences of images, which can be 
played or generated, include any kind of stimuli that it is desired to 
associate with the game, for instance, shaking of a seat, smell, smoke, etc. 
According to a preferred embodiment of the invention, the sound played is 
not necessarily associated with the main image being displayed, but may be 
associated with the value of the input provided by the player. Thus, the 
same image or set of images may be accompanied by different sounds if it 
has been displayed as a result of different inputs from the player. Thus, as 
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will be apparent to a person skilled in the art, a single image may be used to 
generate different sequences, and different sequences may be used to 
generate different scenes, depending on the development of the game. 

The mixing of computer-generated and of real-life images can be extensively 
exploited, because the human mind tends to allow them to blend without 
finding disturbance in them. Thus, in some instances it may be more 
efficient and quick, if the input from the player required too heavy an image 
processing of the image available from the image bank, to generate a 
suitable image, or a part thereof, through the computer, rather than 
utilizing a real-life image. This, for instance, is convenient when a game 
includes gun shots, explosions, or other occurrences requiring special effects. 
Typically, however, the main image and the secondary image are processed, 
as a function of an input provided by a player, so as to create an image 
corresponding to that seen by an observer from a diff erent point of view, or 
so as to zoom in or out of the original image stored in the image bank, 
similarly, to the eyes of the player moving closer of farther away from the 
scene. 

The invention is also directed to a bank of selectable, composite, three- 
dimensional-like image sequences, which are generated by the method of 
the invention. Thus, in one aspect, the invention is directed to a bank of 
selectable secondary sequences of images, suitable to be used as a 
component of a computer game, comprising a main image or a main 
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sequence of images and at least one secondary image or sequence of images 
selectable together with said main image or sequence of images, said 
secondary image or sequence of images having a content that at least 
roughly matches the contents of the main image of said composite, three- 
dimensional-like sequence of images, and said secondary image or sequence 
of images further having been optimized for display together with said main 
sequence of images. 

Without wishing to limit the invention in any way, it should be mentioned 
that in many cases it is preferred to provide images bigger than the screen 
by 25% to 100%, to allow flexibility in the image processing. In such a case, 
the image displayed is only a fraction of the total image, so that portions of 
the image which are not displayed for one desired viewing angle will be 
displayed when the input of the player corresponds to another angle. 

The invention also encompasses an interactive computer game comprising: 

(a) a bank containing a plurality of individually selectable main 
images or main sequences of images, said images being images of 
true-life origin; 

(b) at least one bank of secondary images or sequences of images, the 
contents of which are at least roughly related to the contents of 
said main sequence, said secondary sequences being selectable as 
a function of the selected main images or sequences of images, 
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and each of said at least one secondary image or secondary 
sequence of images having been optionally pre-processed so as to 
optimize its matching with said main image or main sequence of 
images in the selected positioned relationship with it, or to obtain 
other results, as discussed above; 

(c) means for selecting the positioned relationship of each of said at 
least one secondary images or sequences of images and of said 
main images or sequences of images; 

(d) interaction and processing means to select a main image or a 
main sequence of images, as the result of an input provided by a 
player, wherein said selected main image or main sequence of 
images is displayed together with its related secondary image(s) 
or sequence(s) of images, at the same time and in substantially 
the same positioned relationship as selected in step (c) above; 

(e) display means to display said main image or sequence of images, 
together with said at least one secondary image or sequence of 
secondary images, so as to provide a 3D-like feeling to the player; 

(f) means for on-line processing said main image and said secondary 
image, as a function of an input provided by a player; 

(g) means for playing a sound related to said main sequence of 
images and said secondary sequences of images; and 
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(h) optional means for displaying at the same time non-real life, 
computer-generated images in superimposed positioned 
relationship with the main or secondary images. 

According to a preferred embodiment of the invention, the interactive 
computer game comprises means for controlling the speed of display of the 
frames on the display means. According to another preferred embodiment of 
the invention, the means for processing the main image and the secondary 
image, as a function of an input provided by a player, comprise image 
processing means suitable to generate from the image available from the 
image bank an image corresponding to that seen by an observer from a 
different point of view. 

The computer game of the invention preferably, but non-limitatively, 
further comprises means for processing the main image and the secondary 
image, as a function of an input provided by a player, so as to simulate the 
turning of an observer's head. Further suitable means for processing the 
main image and the secondary image, as a function of an input provided by 
a player, comprise image processing means suitable to zoom in or out of the 
original image stored in the image bank. 

The invention is further directed to visual display systems for playing three- 
dimensional-like computer games, comprising display means for displaying 
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a main image or main sequence of images, and at least one additional 
display means for displaying at least one secondary image or secondary 
sequence of images, said display means for displaying a main image or 
sequence of images, and said at least one additional display means for 
displaying at least one secondary image or sequence of images being 
positioned in substantially the same positioned relationship selected for the 
generation of the composite three-dimensional-like image. 



Brief Description of the Drawings 

The above and other characteristics and advantages of the invention will 
be more readily apparent through the following detailed description of 
preferred embodiments thereof, with reference to the appended drawings, 
wherein: 

Fig. 1 schematically illustrates the overlap between three movies to 
be displayed simultaneously; 

Fig. 2 (2A and 2B) illustrates the operation of a game, according to 
two different preferred embodiments of the invention; 

Fig. 3 illustrates the result of a joystick movement simulating the 
turning of an observer's head; and 
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Fig. 4 schematically illustrates the operation involved in image 
processing with head turning, in the case where a larger image than 
displayed is available. 

Detailed Description of Preferred Embodiments 

In order to display an interactive 3d movie, three movies need to be 
prepared as shown in Fig. 1. This means that three films have been taken 
so that the three fields of view overlap to a certain extent, for example 
25% of the field is overlapped in each side. Also, the captured images are 
vertically larger than required, typically in the same amount, say 25% on 
each side. It is very important that the three focal points of the cameras 
coincide, as otherwise the effect of tilting the three cameras together will 
look unnatural and the desired psychophysical effect will not be achieved. 
All this can be done either by using a gig of three rigidly mounted, fixed, 
calibrated cameras, or else, by using the preferred technique of filming the 
three sequences separately and then, using appropriate processing means 
and software, processing them to be fit for display on a 3D-like screen. The 
output of this stage is three image sequences, given on a digital video 
cassette, or, preferably, on a random access data device such as a 
computer hard disk. 

According to a preferred embodiment of the present invention, the three 
separate movies of Fig. 1 above are merged into one "large" (broad view) 
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movie, using mosaic techniques. Thus, the resulting large movie is 
equivalent to a movie taken by using the same camera. Similarly, the 
large movie may be stored on a digital video cassette, or, preferably, on a 
random access data device such as a computer hard disk, or on any other 
suitable media. 



Once this is ready, an interactive 3d movie can be presented in the 
following way. Reference is now made to Figure 2 A. In the preferred 
embodiment shown in this figure, we have three image sequences residing 
in three computer disks, 211, 212, 213. The main computer 230 pulls 
sequentially the three sequences, and prepares them to be displayed on 
the three corresponding display means 241, 242, 243, which can be video 
screens, video projectors, LCD screens, or any other display means. It is 
important the display means will reproduce the geometry of the three 
cameras, as far as the geometry of the viewing eyes in relation to the 
displays will be similar (in the geometrical sense) to the geometry of the 
three (real or virtual) cameras. It should be kept in mind that the original 
image sequences have been taken with a field of view which is larger than 
that required to be properly displayed on a 3d-like display means. A 
computer interaction means 220 is provided, such as a mouse or a joystick, 
to allow interaction of the user with the system. 
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Fig. 2B illustrates the operation of a game according to a preferred 
embodiment of the present invention, using one large movie. The large 
movie is stored in a computer disk 280. The main computer 270 extracts 
the large movie sequence information and generates three sub -sequences, 
which size and relative location are a function of the desired position, 
which may be the center or any other position selected by the player. The 
main computer manipulates the side movie sequence(s) to represent the 
correct angles of the camera. Similar computer interaction means 260 is 
provided, such as a mouse or a joystick, to allow interaction of the user 
with the system. 

This control mechanism provides parameters that effect the way that the 
images are displayed on the screen. In this particular embodiment the 
parameters are horizontal and vertical tilt (viewing angle), camera zoom, 
and motion speed. It is clear to anyone skilled in the art that many other 
parameters and parameter combinations can also be controlled or coupled 
to input devices, such as sound volume, chair wobble, mechanical effects, 
smell, digital video effects such as fog and many more. The following is a 
detailed description of how the image is prepared to be displayed, using 
the controls given by the interaction means, in this case horizontal tilt 
theta, vertical tilt phi (given in radians), zoom z (given as percentage of 
default image which is, say 70% of full image), and motion speed t (given 
as percentage of standard speed - 25 or 30 frames per second). First the 
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speed is controlled by standard video techniques, such as dropping frames 
and/or generating virtual frames to be displayed with or without original 
frames. For instance, virtual frames are generated in case when speed 
reduction is required, e.g., creating an intermediate frame interpolating 
between two existing frames (e.g., frame No. 1.5, interpolating midway 
between frames No. 1 and No.2). Zoom is controlled by clipping from the 
original image a larger or smaller sub-image and enlarging it to the 
required image size. In addition, zoom may induce compensations like, for 
instance, translation on secondary sequences. Rotation is done by the 
standard application of projective transformations, where a horizontal 
rotation of an image by theta is computed by the transformation: 

x v = ( x cos(theta) - sin(theta) ) / (x sin(theta) + cos (theta) ) 
y v = y / (x sin(theta) + cos (theta) ) 

while vertical rotation by phi is given by the transformation: 
x y = x / (y sin(phi) + cos (phi) ) 

y % - ( y cos(phi) - sin(phi) ) / (y sin(phi) + cos (phi) ) 

All in all, the final transformation involving zoom, vertical, and horizontal 
rotation is obtained by multiplying together the separate transformations, 
and then applying the result to the image. This can be done either by a 
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standard computer utilizing a fast CPU module such as a "Pentium Pro" 
which is sold by Intel Corporation, or by a computer assisted by 
specialized graphics hardware accelerators (video texture mapping 
module), such as an 02 workstation marketed by Silicon Graphics 
Limited, California, U.S.A.. 

Looking now at Fig. 3, four sequences of images are given (Fig. 3A through 
Fig. 3D), illustrating consecutive stages in the on-line image processing 
carried out during the game, as the result of an input by the player, 
corresponding to the turning of the head of the player to the left. Three 
images are shown, which, for illustration purposes are shown on a plane, 
but which should be understood to be positioned at an angle, e.g., as 
schematically shown in Fig. 1. The images are numbered 1-3, from right to 
left. 

Each of the images 1, 2 and 3 , recovered from the data bank, contain 
more information than displayed. However, each image is masked so that 
only a portion thereof is shown. Thus, for instance, image (1) of Fig. 3A is 
taken with a broader field of view, which includes the details not seen in 
Fig. 3A, but which are seen in image (1) if Fig. 3D. Similarly, the details 
seen in the right hand side of image (1) of Fig. 3D are contained in the 
image (1) of Fig. 3A, but they are masked and therefore not seen. In this 
way, it is possible to provide an effect of change and movement, while 
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moving with the mask within the same image, without making any 
further processing of the image, except the masking. Before being cut or 
masked, the images are rotated according to the aforementioned (or 
alternative) formulae, to provide the effect of turning. 

This procedure is further schematically illustrated in Fig. 4. The whole 
image available from the date bank is seen, together with the mask (the 
inner square) which shows the part of the image which is being displayed. 
In Fig. 4(a) the leftmost part of the building is displayed, and the portions 
of the building found outside the mask are not seen on the display means. 
A the head of the observer moves to the left (as a result of an input given, 
e.g., by a keyboard, a joystick, or other means) the mask also rotates to the 
left, showing portions of the building formerly not displayed, and hiding 
the earlier displayed leftmost parts of the building. 

Turning again to Fig. 3, it should be remembered that while the 
movement of the head is effected, the game continues and the frames 
continue to be displayed, at the required speed. This is seen, for instance, 
by looking at image (3). In Fig. 3A the moving car is not only much more 
to the right than in Fig. 3D, but is also not as close to the observer. While 
the observer's head moved, the car has continued to advance. 
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Of course, this example and the above description has been provided only 
for the purpose of illustration, and are not intended to limit the invention 
in any way. As will be appreciated by the skilled person, the invention can 
be carried out in a great variety of ways, employing many different image 
processing techniques and apparatus, and using different images and 
images obtained and generated in different ways, all without exceeding 
the scope of the invention. 
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Clairns 

L A method for generating a composite, three-dimensional-like image 
sequence, by the interactive assembly and processing of two or more 
separate image sequences, comprising the steps of: 

(a) providing a bank containing a plurality of individually selectable 
main images or main sequences of images, said images being 
images of true-life origin, and said images or sequences being 
shot separately, or extracted from one source; 

(b) providing at least one bank of secondary images or sequences of 
images, said images or sequences being shot separately or 
extracted from one source, the contents of which are at least 
roughly related to the contents of said main sequence, said 
secondary sequences being selectable as a function of the selected 
main images or sequences of images; 

(c) selecting the positioned relationship of each of said at least one 
secondary images or sequences of images and of said main images 
or sequences of images; 

(d) pre-processing each of said at least one secondary image or 
secondary sequence of images so as to produce a desired 
processed image; 

(e) providing selecting means to select a main image or a main 
sequence of images, as the result of an input provided by a player, 
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wherein said selected main image or main sequence of images is 
displayed together with its related secondary image(s) or 
sequence(s) of images, at the same time and in substantially the 
same positioned relationship as selected in step (c) above; 

(f) providing means for on-line processing said main image and said 
secondary image, as a function of an input provided by a player; 

(g) optionally, playing a sound related to said main sequence of 
images and said secondary sequences of images, and/or playing or 
generating additional outputs related to such sequences of 
images; and 

(h) optionally, displaying at the same time non-real life, computer- 
generated images in superimposed positioned relationship with 
the main or secondary images. 

2. A method according to claim 1, wherein the main image and the 
secondary image are processed, as a function of an input provided by a 
player, so as to create an image corresponding to that seen by an observer 
from a different point of view. 

3. A method according to claim 1, wherein the main image and the 
secondary image are processed, as a function of an input provided by a 
player, so as to zoom in or out of the original image stored in the image 
bank. 
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4. A method according to claim 1, wherein the main image and the 
secondary image are processed, as a function of an input provided by a 
player, so as to simulate the turning of an observer's head. 

5. A method according to claim 1, wherein the main image and the 
secondary image are processed, as a function of an input provided by a 
player, so as to change the speed of movement of the observer. 

6. A method according to claim 1, wherein the sound played is not 
associated with the main image being displayed, but is associated with 
the value of the input provided by the player. 

7. A method according to claim 4, wherein the image displayed is Only a 
fraction of the total image, so that portions of the image which are not 
displayed at one time are displayed when the input of the player 
corresponds to a turning of the observer's head. 

8. A method according to claim 7, wherein the size of the image is 25% to 
100% larger than the displayed image. 

9. A bank of selectable, composite, three-dimensional-like image sequences, 
whenever generated by the method of claim 1. 
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10. A bank of selectable secondary sequences of images, suitable to be used 
as a component of a computer game, comprising a main image or a main 
sequence of images and at least one secondary image or sequence of 
images selectable together with said main image or sequence of images, 
said secondary image or sequence of images having a content that at 
least roughly matches the contents of the main image of said composite, 
three-dimensional-like sequence of images, and said secondary image or 
sequence of images further having been optimized for display together 
with said main sequence of images. 



11. An interactive computer game comprising: 

(a) a bank containing a plurality of individually selectable main 
images or main sequences of images, and said images or 
sequences being shot separately, or extracted from one source, 
said images being images of true-life origin; 

(b) at least one bank of secondary images or sequences of images, the 
contents of which are at least roughly related to the contents of 
said main sequence, and said images or sequences being shot 
separately, or extracted from one source, said secondary 
sequences being selectable as a function of the selected main 
images or sequences of images, and each of said at least one 
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secondary image or secondary sequence of images having 
optionally been pre-processed; 

(c) means for selecting the positioned relationship of each of said at 
least one secondary images or sequences of images and of said 
main images or sequences of images; 

(d) interaction and processing means to select a main image or a 
main sequence of images, as the result of an input provided by a 
player, wherein said selected main image or main sequence of 
images is displayed together with its related secondary image(s) 
or sequence(s) of images, at the same time and in substantially 
the same positioned relationship as selected in step (c) above; 

(e) display means to display said main image or sequence of images, 
together with said at least one secondary image or sequence of 
secondary images, so as to provide a 3D-like feeling to the player; 

(f) means for processing said main image and said secondary image, 
as a function of an input provided by a player; 

(g) means for playing a sound related to said main sequence of 
images and said secondary sequences of images; and 

(h) optional means for displaying at the same time non-real life, 
computer-generated images in superimposed positioned 
relationship with the main or secondary images. 
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12. An interactive computer game according to claim 11, comprising means 
for controlling the speed of display of the frames on the display means. 

13. A computer game as claimed in claim 11, wherein the means for 
processing the main image and the secondary image, as a function of an 
input provided by a player, comprise image processing means suitable 
to generate from the image available from the image bank an image 
corresponding to that seen by an observer from a different point of view. 

14. A computer game according to claim 11, comprising means for 
processing the main image and the secondary image, as a function of an 
input provided by a player, so as to simulate the turning of an observer's 
head. 

15. A computer game as claimed in claim 11, wherein the means for 
processing the main image and the secondary image, as a function of an 
input provided by a player, comprise image processing means suitable 
to zoom in or out of the original image stored in the image bank. 

16. A visual display system for playing three-dimensional-like computer 
games, comprising display means for displaying a main image or main 
sequence of images, and at least one additional display means for 
displaying at least one secondary image or secondary sequence of 
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images, said display means for displaying a main image or sequence of 
images, and said at least one additional display means for displaying at 
least one secondary image or sequence of images being positioned in 
substantially the same positioned relationship selected for the 
generation of the composite three-dimensional-like image. 

17. A method for generating a composite, three-dimensional-like image 
sequence, by the interactive assembly and processing of two or more 
separate image sequences, essentially as described and illustrated. 



18. An interactive computer game, essentially as described and illustrated. 
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